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Typical performance of low-density parity-check (LDPC) codes over a general binary-input 
output-symmetric memoryless channel is investigated using methods of statistical mechanics. Theo- 
retical framework for dealing with general symmetric channels is provided, based on which Gallager 
and MacKay-Neal codes are studied as examples of LDPC codes. It has been shown that the ba- 
sic properties of these codes known for particular channels, including the property to potentially 
saturate Shannon's limit, hold for general symmetric channels. The binary-input additive-white- 
Gaussian-noise channel and the binary-input Laplace channel are considered as specific channel 
noise models. 



I. INTRODUCTION 



II. THE GENERAL FRAMEWORK 



We investigate the typical performance of low-density 
parity-check (LDPC) codes over a general binary-input 
output-symmetric (BIOS) memoryless channel. Previous 
statistical physics based analyses of LDPC codes have 
discovered some interesting properties, including the fact 
that they can, in principle, saturate the information- 
theoretic upper bound (Shannon's bound defined by 
the channel coding theorem 0]) with low connectivity 
values. Existing statistical mechanical studies on the 
LDPC codes, however, have been mostly confined to 
the case of binary symmetric channel (BSC), which fits 
into the statistical-mechanical framework in a natural 
way (2| ||, 0, ||]. Notable exceptions are the work by 
Montanari || that discusses the case of binary-input 
additive-white-Gaussian-noise channel (BIAWGNC) as 
well as the BSC case and the study of Sourlas codes 0, 
a simple LDPC code, in which non-BSC channels are ad- 
dressed |8|, ||, |l0| . From the statistical- mechanical point 
of view, LDPC codes are regarded as random spin sys- 
tems; it is therefore natural to expect that they will ex- 
hibit some sort of universality, just as typical statistical- 
mechanical systems do, so that general properties of 
LDPC codes observed in the BSC case will be preserved 
when different communication channels are considered. 
In this paper we investigate the properties of LDPC 
codes in binary-input output-symmetric channels and 
show that this is generally the case. In particular, we 
show that the finite connectivity LDPC codes can satu- 
rate Shannon's bound for general BIOS channel. 

The paper is organized as follows: In section II we 
introduce the general framework, notation, codes and the 
channels that we will focus on. In section III we will 



briefly describe the calculation for the various channels, 



while the results obtained will be described in section IV 
followed by the conclusions. 



A. Symmetric channels 

We consider the general class of binary-input output- 
symmetric (BIOS) memoryless channel. The input of the 
channel is binary (±1), and the output may take any real 
value. The characteristics of a channel is described by the 
channel transition probabilities, P(y\x = 1) and P(y\x = 
— 1). Let p(y) = P{y\x = 1). A symmetric channel is 
characterized as a channel whose transition probabilities 
satisfy P{y\x = —1) = P(— y\x = 1) = p{—y). Various 
types of channel models of practical interest fall into the 
class of BIOS channels, including the binary symmetric 
channel (BSC) 



PBsc(y) = (l -p)5(y- !) 1), 



(1) 



the binary-input additive-white-Gaussian-noise channel 
(BIAWGNC) 



Pbiawgnc (y) 
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and the binary-input Laplace channel (BILC) 



PBiLc(y) = j e 



-|y-l|/A 



(2) 



(3) 



Each of the parameters p, a 2 , and A represents the degree 
of degradation induced by the channel noise. We call each 
of them the noise level and let d denotes the generic one. 

An apparent technical difficulty in dealing with a gen- 
eral channel of real-valued output is that it is not at all 
obvious how to define the syndrome from the received 
signal: The modulo 2 arithmetic involved in computing 
syndrome in the BSC case is not directly applicable to 
the cases of real- valued received signal. This difficulty is 
resolved by using a truncation procedure fill] : We con- 
ceptually consider another fictitious binary-input binary- 
output channel in addition to the channel under study. 
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Let r be the (fictitious) output symbol of this fictitious 
channel. We can assign to r either of the values ±1 arbi- 
trarily, and the binary channel noise ( for the fictitious 
channel is defined therefrom, via r = x(. For the sake 
of making the argument simple, we assign r = 1 without 
loss of generality. Since the prior probability of £ (be- 
fore receiving y) should be P(( = ±1) = 1/2, the joint 
distribution of y and ( is given by 



P(y,0 



(4) 



since the truncation procedure used here yields x = £• 



where C s and C n are sparse matrices of dimensionality 
M x N and M x M , respectively; C„ is assumed invert- 
ible. The information rate for the code is R = N/M for 
unbiased message. 

In regular MN codes the row and column constraints 
are imposed on both matrices C s and C n . The number 
of non-zero elements per row of C s and C n should be ex- 
actly K and L, respectively. Also here, we do not discuss 
irregular MN codes |l5| ] in this paper. The number of 
non-zero elements per column of C s and C n are set to C 
and L, respectively, where C — KM/N holds. We call 
the resulting code a (K, C, L)-MN code. 



B. Gallager code 

LDPC codes have been originally introduced by Gal- 
lager in his seminal work from 1963 fl2| . Gallager's orig- 
inal construction |l^] is one of the most extensively stud- 
ied LDPC codes in the information theory literature. It 
is defined by its parity-check matrix A = [C1IC2] of di- 
mensionality (M — N) x M, which is taken to be random 
and very sparse. The submatrix C2, of dimensionality 
(M — N) x (M — N), is assumed invertible. 

In the encoding step, the encoder computes a codeword 
from the information vector £ £ {0, 1}^ by employing a 
generator matrix G 



x = G T $, mod 2, 
where the generator matrix is defined by 
G = [IIC^Ci] mod 2. 



(5) 



(6) 



This construction ensures AG T = mod 2. The infor- 
mation code rate for unbiased messages is R = N/M. 

In regular Gallager codes, the number of non-zero ele- 
ments per row of A is fixed to be K. We call it the row 
constraint. Average number of non-zero elements per col- 
umn is then C = K(M — N) /AI, whereas we will consider 
the case in which the number of non-zero elements in each 
column is forced to be exactly C, which we term the col- 
umn constraint. Irregular Gallager codes can be defined 
by relaxing these constraints. It has been known that 
making code construction irregular may improve perfor- 
mance significantly |l3| , but we will not discuss irregular 
codes in the current paper. We call the resulting regular 
Gallager code a (C, iT)-Gallager code. 



C. MN code 

We also discuss a variant of LDPC codes, called the 
MN code @, |J . The generator matrix G T of the MN 
code is defined by 



III. ANALYSIS 

A. Gallager code 

The basic idea behind the statistical-mechanical treat- 
ment of the LDPC codes is the equivalence between the 
decoding problem and the thermal equilibrium distribu- 
tion of a dilute Ising spin system. In order to see this 
in the Gallager code case, one should first note that the 
decoding problem is to find r which is best supported 
(i.e., most probable) by the received signal y among the 
set of t satisfying the parity-check equation (AC, = At 
mod 2 if we write it in the {0, l}-notation). The set is 
expressed as 



lim exp 

7 — >oo 



n rj-l) =l}, 



where 



CO*) = {j\A M = 1} 



(8) 



0) 



denotes the set of indices for which the parity-check ma- 
trix A has l's in /n-th row, and 



j»= n 



(10) 



is fi-th check. The posterior probability of r condi- 
tioned on the received signal y then acquires the following 
Gibbs-Boltzmann form: 

P-r(T\y) = |exp[-/3H 7 (x;y,J)] (11) 

in which we have to take the limit 7 — > 00 and consider 
it at = 1 (Nishimori's temperature f§, [If}, |l7|, |l8)) 
in order to obtain the true posterior. The Hamiltonian 
7i 7 (x;y, J) is defined as 



W 7 (r;y, J) 



G T = C- X C S mod 2, 



(7) 



M-N , 

-7 e U n 

M 

-^logpfoy,-), 
i=i 



^-1 



(12) 
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The channel characteristics enters into the Hamiltonian 
as the term logp(r,'t/j) which, by noting that Tj takes ±1, 
can be rewritten as 



1 



\o g p{T j y j )=r j -\o g ^- ) 



PM- + llo g p( yj )p(-yi)- (13) 
A yj) z 



From this expression it immediately follows that it is the 
log-likelihood ratio hj = (l/2)\og(p(yj)/p(~yj)) of the 
channel noise yj which serves as the external field acting 
on site j, and that the channel characteristics defines the 
field distribution. Analyzing the effect of having different 
communication channels on the code properties, there- 
fore reduces to investigating the effect of different field 
distributions on the physical properties of the system. 
The field distributions p(h) for various channel models 
are as follows: 



• BSC: 

PBSc(h) 



(l-p)6[h-llog — 
2 p 



pS [ h + i log — 
2 p 



(14) 



BIAWGNC: 



PBIAWGNCW = \ a —e-^°-^l^ 
V Z7T 



(15) 



• BILC: 

PBILCW 



-5(h-\- 1 ) + 



-2\~ 



Sih + X- 1 ) 

1 



+ <h< A- 1 ]-e' l - A 



(16) 



where x[A] is the indicator function, taking 1 when 
X is true and otherwise. 

Sketches of these field distributions are given in Fig. [l] 

We assume that the free energy of the system is self- 
averaging, that is, 

/ = -4 M- l Qo e Z) AtV , (17) 

and evaluate the average (-}A,y over the received signal y 
and the randomness of the parity-check matrix A using 
the replica method, 



1 o 
/ = -- lim limM- 1 — log(Z n ) A ,„. 

(3 M^oon^O dn 



(18) 



In calculating the free energy, we perform the gauge 
transformation Tj — > Cj T j-> yj — > CjVj- The average over y 
can be taken with respect to Iljli P(Vj) after having per- 
formed the gauge transformation. We have to introduce 
a random tensor to take average over A. 



1-p' 






f 


p 









(a) BSC 




1/(7 2 

(b) BIAWGNC 




-MX MX 

(c) BILC 

FIG. 1: Field distributions corresponding to various BIOS 
channels. 



Following basically the same procedure as in j3j and 
exchanging the order of the two limits, taking the limit 
M — > oo first, one obtains 



1 , 9 
f = — — lim — — Extr 

P n—>0 On q.q 



where 



Qx{q)- G^(q,q) +G^{q) 



, (19) 



Gi(q) = log XI E <?£..• Qm -™ log 2, 

m=0 (ai---cK m ) 

n 

S2(q,q) = E E ia 1 -a m Q.a 1 -a m , 
m=0 (ai---a m ) 



G 3 {q) = log 



e (n^) 



\Q = 1 



^(E E 



-0 (o>i ■■■a.m ) 



(20) 
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To proceed further we adopt the replica-symmetric 
(RS) ansatz and let 

Qai-a m = qo Ju rn ir(u)du, q ai - am = qo Ju m n(u)du. 

(21) 

We will use the following simplifying notation. 

K 

ir K (u) du = tt(wj) duj 
The replica-symmetric free energy / becomes 



/ RS = -Extri — log 2 



(22) 



+ C log(l + uu) 7r(u) n(u) du du 



f / log(l + n^)7r K (n)d« 

J 3=1 

c c 
f (log p(y) Y[(i + ui) + p(-y) 



x 7t c (m) d« 



(23) 



in which go arL d <?o have been eliminated using the extrem- 
ization condition qoqo = C. Heuristic construction of a 
sufficient condition to the extremization problem with re- 
spect to 7r and 7r is possible, and it gives the following 
saddle-point equations. 



7r(u) = J (^5 u — tanh^fo(y) + ^""^ tanh -1 ui 

x 7r c ' _1 ('u) du 
tt(u) = J 5^u— Y[ Uj^j tt k ^ 1 (u) du 



We have let 



2 p(-y) 



(24) 



(25) 



The performance of the code is quantified by the overlap 
rn = M^ 1 J^kLi (j( T j)i which is given as 



where 
P{z) 



ui = j sign(z) P{z) dz, 
J (^5 z - tanh^fo(y) + ^ tanh" 1 



(26) 



x tt c (u) du. 



V 

(27) 



B. MN code 

The decoding problem for the MN code is to find S and 
r which are the best suitable in view of the received signal 
y among the sets of S and r satisfying the parity-check 
equation (C S S + C n T = C s £ + C n C mod 2 if written in 
the {0, l}-notation). Defining the /zth component of the 
check J as 



n & n ( 2g ) 

jec a (n) iec n (p.) 

where 

r,(A*) = {j\(c.U = i}, = {i\(Cn)„i = i}, 

the posterior probability of S and r conditioned on the 
received signal y and the check J is given by 

P 7 (S,r\y,J) = lexp[-/3W 7 (S,T;y, J)], (30) 

in the limit 7 — > 00 and at [3 = 1 , where the Hamiltonian 
TL 1 {S, t; y, J) is defined as 

^(s.rjtf.j) = -te(^ n s i n t '- x ) 

iV M 

-F.^-^logpfaw), (31) 

3=1 1=1 

where F s is a parameter representing the bias of the in- 
formation vector £ in such a way that P(£j = ±1) = 
(l±tanhF s )/2 holds. The form of Eq. @ clearly shows 
that the channel characteristics again acts as random 
field on {t;}, where the log likelihood ratio gives the ac- 
tual value of the field. 

The replica calculation can be done along the same way 
as in the case of the Gallager code. We have performed 
the gauge transformation Sj — > £jSj, Tj — > Cj T ji an d 
Uj — ► (jTj. The free energy / becomes 



1 . d 
f = — — Una -7— Extr 

p n^O On q,q,r,r 



(32) 



where 



Gi{q,r) 



G2(q,q,r,f) 



lo sE E 

m=0 (o r .. 0m ) 

— nlog2, 



^ ^ qa 1 ---a m q ai ---a rl 

m=0 (ai---a m ) 

+ ^E E 



-0 (ax ■ ■ -a m ) 



(33) 
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and 



g 3 (q,f) = log 



E 



LS 1 ,...,S" 1 



m=0 (a,-a m ) 

E (n^ Q y) 



LT 1 ,...,T" 



zt(E E ^--^ 



-0 (q 1 ■ ■ -Q m ) 



(34) 



We adopt the RS ansatz as before, under which we 
have 



r / v m p(v)dv, f ai ... am =f v m p(v)dv 



(35) 

in addition to Eq. (|2l|). The replica-symmetric free en- 
ergy / becomes 



f BS = 



1 c 

- Extr \ - log 2 



+ C yy log'(l + mm) 7r(u) 7r(u) tiit (iu 
+ (I log(l + w) p(v) p{v) dv dv 



• v fe=i ;=i 

x 7T (it) ciw p L (t>) dv 

c 



/(log £ e« s H( 1 + ^ 

" 7? / \ log E p( T y)II( 1 + r?} *) 

" \ L~ LI 7 1 



7r c '('u) du 



T=±l 



x / 5 L ('u) (it) >, 



(36) 



in which go, go, ro, and fo have been eliminated using 
the extremization conditions, go<?o = C and fo^o = L. 

Construction of a heuristic solution to the extremiza- 
tion problem can be done in the same manner, which 



yields the following saddle-point equations: 

c-i 



7r(u) = 

7r(tt) = 

p(v) = 

p(v) = 



J (d u- tanh^F s £ + ^ tanh 1 



ui 



x 7r c 1 (m) du 

K-l L 



II sfu- [J u k f[vi\ n K - 1 (u)dup L (v)dv 
J v- tanh^(y) + ^ tanh -1 



x p L ~ 1 (v)dv 

K L-l 



p L -\v)dv 



The overlap is then evaluated by 

m = J sign(z) P{z) dz, 

where 

P(z) = J(s z- tanh ^-Fs^ + E tanh_1 
x 7t c (m) 



(37) 



(38) 



ui 



e 

(39) 



It is worthwhile mentioning that, when the message 
is unbiased (F s = 0) and K is even, saddle-point 
solutions have the following symmetry: For each so- 
lution {%(u),n(u),p(v),p(v)} there is another solution 
{tt(— u), 7r(— u), p(v),p(v)}. The latter has the same over- 
lap as that of the former with the opposite sign. 



IV. RESULTS 



A. Gallager code 



1. Analytical solutions 



Of particular interest is the ferromagnetic state, which 
corresponds to an error-free communication. One can see 
that the assertion 



7r(u) = 6(u — 1), tt(u) = S(ii — 1) 



(40) 



always satisfies the saddle-point equation (gj) irrespec- 
tive of the values of K and C (provided that K, C > 2), 
thereby providing a solution. The overlap and the free 
energy of the solution at (3 = 1 are mf cr ro = 1 and 
/fcrro = — 0-°SP(y))yt respectively. One can therefore 
identify this as the ferromagnetic solution. 

Another solution, which can be found in the limit K — > 
oo, is the sub-optimal ferromagnetic solution 

7r(u) = (%-tanh%)]) y , tt(u) = 6(u), (41) 
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for which 

m sf = (sign[p(y) - p{-y)] ) y (42) 

and 

/ sf = ^log2- (\og[p(y)+p(-y)]) y . (43) 
The difference of the free energy is expressed as 

/ S f-/fcrro = C-i?l0g2, (44) 

where C is the channel capacity of the BIOS channel de- 
fined as 

C = log 2 - <bg[p(y) +p(-y)]) y + {\ogp{y)) y . (45) 

This proves that the thermodynamic transition between 
the ferromagnetic and sub-optimal ferromagnetic solu- 
tions (no other solution has been identified in this case) 
occurs at the theoretical limit. This means that the max- 
imum rate i? max , up to which error-free communication 
is theoretically possible, asymptotically achieves the the- 
oretical limit as K — > oo. This result has been known 
for BSC channel j|, |) in the physics literature and is in 
agreement with results reported in the information the- 
ory literature ]Tl| ] . The current result is an extension to 
the case of a general BIOS channel. 

2. Numerical solutions of saddle-point equations 

In finite- K cases no simple analytical solution exists 
other than the ferromagnetic one, so one has to solve 
the saddle-point equations numerically. We have done it 
for BIAWGNC and BILC. The dependence of the over- 
lap m on the noise level d {a 2 for BIAWGNC, and A 
for BILC) is qualitatively the same as that observed in 
BSC: For K > 3 the ferromagnetic solution is locally 
stable over the whole range of noise levels. At d = d s , 
another solution with m < 1 appears, which defines the 
spinodal point. At a higher noise level d — dt > d s 
thermodynamic transition takes place, beyond which the 
ferromagnetic solution with m — 1 becomes metastable 
(see Fig. ||). Table | summarizes the results for the BI- 
AWGNC case, showing the spinodal point a 2 (the value 
of the variance at which new, non ferromagnetic, solu- 
tions emerge), the thermodynamic transition point a 2 
(at which the thermodynamic transition occurs), and <7g, 
the information-theoretic upper bound of the variance 
allowing error-free communication. 

Table |l| summarizes the results for the BILC case, 
showing the values of the spinodal point A s , the ther- 
modynamic transition point At, and the information- 
theoretic upper bound Ao- 

It should be noted that the results for the spinodal 
point agree well with the results obtained by the density 
evolution approach G3], as expected, since the saddle- 
point equations by the replica analysis happen to coincide 
with the time evolution equations in the density evolu- 
tion. 



m 
1 - 




FIG. 2: Noise-overlap diagram for Gallager code. Thick solid 
lines stand for the stable state, thin solid lines for metastable 
state, and broken lines for unstable states. The ferromagnetic 
solution is characterized by the m = 1 solution, while m < 1 
defines the suboptimal ferromagnetic solution. 

TABLE I: The variances of and of at the spinodal point and 
thermodynamic transition, respectively, for the BIAWGNC 
for various code parameters; <tq, denoting the information- 
theoretical upper bound for error-free communication, is also 
shown. 
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B. MN code 

1. Analytical solutions 

In the following we restrict our discussion of the MN 
code to the unbiased case F s = 0. The ferromagnetic 



TABLE II: The parameter values X s and A t at the spin- 
odal point and thermodynamic transition, respectively, for 
the BILC with various code parameters; Ao, denoting the 
information-theoretical upper bound for error-free communi- 
cation, is also shown. 
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solution, corresponding to the error-free communication, 
can be constructed for the MN code with L > 2. (In 
fact, in the case L — 1 the matrix C„ reduces to a simple 
permutation matrix, so that we have to estimate each 
element of noise separately. This case is not at all inter- 
esting and therefore we will not discuss it any more.) It 
is given by 

■k(u) = 5{u — 1), tt(u) = 5(u — 1), 

p(v) = S(v-l), p(v) = S(v-l), (46) 

for which mf erro = 1 and 

/ferro = -|(l0gKy)> a - (47) 

The MN code has the following paramagnetic solution 
for K > 2: 

tt(u)=S(u), 7t(u) = S(u), 

p(v) = (S[v-t a ahh(y)]) y , m = 8(v), (48j 

which yields m para = and 

/para = (f " *) Iog2 - -|(log[p(|,) + ?(-»)]>„. (49) 

Again, since 

/para " /fcrro = ^ (C - R log 2) (50) 

holds, we conclude that for the MN code the maximum 
rate i? ma x, theoretically allowing error-free communica- 
tion, achieves the theoretical limit as long as K > 2, 
L > 2, provided that there is no locally stable solu- 
tion other than the ferromagnetic and paramagnetic so- 
lutions. This result is an extension of the result reported 
in 0, |^] to the case of a general BIOS channel. 

It should be noted that the paramagnetic solution J46| ) 
is also valid in the limit L — > oo for the case K = 1. This 
means that the above-mentioned result also holds for the 
case of K = 1 asymptotically in the limit L — > oo. 

2. Numerical solutions of saddle-point equations 

In order to explore solutions other than the ferromag- 
netic and paramagnetic solutions, we have to solve the 
saddle-point equations numerically. We have done it for 
the BIAWGNC and BILC cases. We observed qualita- 
tively the same characteristics as those reported in || . 

The obtained numerical results suggest that the quali- 
tative physical properties are categorized into three types 
according to the K value: cases with K = 1, K = 2 and 
K > 3, whereas it is only affected quantitatively by the 
values of C and L, as described in the following. 

The structure of noise-overlap diagram for the MN 
code with K = 1 is qualitatively the same as that for 
Gallager code (see Fig. 0): At very low noise level only 



TABLE III: The variances cf, erf, and a\ at the spinodal 
point and thermodynamic transition, and at bifurcation of 
paramagnetic solution, respectively, for (K, C, L)-MN codes 
over the BIAWGNC and various code parameters; ctq, de- 
noting the information-theoretical upper bound for error-free 
communication, is also shown. 
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TABLE IV: The parameter values X s , At, and Xt at the spin- 
odal point and thermodynamic transition, and at bifurca- 
tion of paramagnetic solution, respectively, for (K, C, L)-MN 
codes over the BILC and various code parameters; Ao, de- 
noting the information-theoretical upper bound for error-free 
communication, is also shown. 



K 


C 


L 


R 


A s 


A t 


A b 


Ao 


1 


2 


3 


0.5 


0.652 


0.714 




0.752 


1 


2 


4 


0.5 


0.619 


0.740 




0.752 


1 


2 


5 


0.5 


0.582 


0.748 




0.752 


1 


3 


2 


0.333 


0.903 


0.934 




1.055 


1 


3 


3 


0.333 


0.831 


1.040 




1.055 


1 


3 


4 


0.333 


0.735 


1.051 




1.055 


2 


3 


2 


0.667 


0.525 


0.551 


0.597 


0.553 


2 


3 


3 


0.667 


0.464 


0.553 


0.493 


0.553 


2 


3 


4 


0.667 


0.419 


0.553 


0.437 


0.553 


2 


4 


2 


0.5 


0.689 


0.751 


0.771 


0.752 


2 


5 


2 


0.4 


0.807 


0.914 


0.894 


0.914 


the 


ferromagnetic 


solution with m = 


1 exists. 


At a 



certain noise level d = d s another metastable solution 
with m < 1 appears, and it becomes dominant beyond 
d — d t > d s . Since the latter solution is obtained only nu- 
merically, there is no guarantee that the thermodynam- 
ical transition dt is equal to the information-theoretical 
limit do. Numerical results show that in general d t is 
smaller than do: However, it is also observed that, for 
fixed C, increasing L makes d s smaller and dt larger, the 
latter of which approaches the information-theoretical 
limit do as L — + oo, as discussed at the end of the previ- 
ous subsection. Even for finite L the value of dt may be 
numerically very close to do, especially when the rate R 
is small. These properties have already been reported for 
the BSC case ||, so that our finding implies that they 
also hold for the BIAWGNC and BILC cases, revealing 
some sort of universality. 

The noise-overlap diagram for the cases with K = 2 
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m 



m 



















d t 





(a) 



m 















d s 






(b) 



FIG. 3: Noise-overlap diagram for the cases with K = 2. 

has the general structure shown in Fig. ^. The diagram 
is characterized by three transition points: the spinodal 
point d s , the thermodynamic transition dt , and the bifur- 
cation point db- The order of the thermodynamic transi- 
tion dt and the bifurcation point db varies with the values 
of C and L, so that the bifurcation pattern for the cases 
with K = 2 is further divided into two sub-categories de- 
pending on the order of the two transitions: d s < db < d t 
for the first group, and d s < d t < db for the second 
group. The noise-overlap diagrams for these groups are 
illustrated in Fig. || (a) and (b), respectively. By the local 
stability analysis the bifurcation point db is determined 
by 



v P {v)dv = (c- ir i/L , 



(51) 



which allows us to decide the type of bifurcation of a par- 
ticular case. See the appendix for derivation of Eq. (51). 
As a result, we found that only a few cases with small 
values of C and L fall into the second category. The 
values of C and L for which the (2,C, L)-MN code fall 
into the second category depend on the channel char- 
acteristics; as far as we have observed, only the cases 







FIG. 4: Noise-overlap diagram for the cases with K > 3. 



with L — 2 fall into the second group. For the BI- 
AWGNC case, the (2, 3, 2)-MN code is the only one in- 
stance, whereas for the BILC case, both (2, 3, 2)- and 
(2,4, 2)-MN codes belong to this group. (For the BSC 
case, (2, 3, 2)-, (2, 4, 2)-, and (2, 5, 2)-MN codes belong to 
this group.) All the (2, C, L)-MN codes but those men- 
tioned above are in the first group. For the cases in the 
second group, the thermodynamic transition dt must be 
less than the information-theoretic limit do: However, it 
turns out numerically that d t is very close to do. 

We observed that the noise-overlap diagram for the 
cases with K > 3 is relatively simple for the BIAWGNC 
and BILC cases, just as in the BSC case (Fig. ||): The 
ferromagnetic solution with m = 1 (and its mirror im- 
age with m = — 1 when K is even) and the paramag- 
netic solution are the only stable solutions found, both 
of which are locally stable over the whole range of the 
noise level. The system exhibits a first-order transition 
at the information-theoretic limit d t . We did not find any 
solutions other than the ferromagnetic and paramagnetic 
solutions. 



V. CONCLUSIONS 

We have analyzed typical performance of LDPC codes 
over BIOS channel using statistical mechanics. We have 
shown for the case of LDPC codes that the log-likelihood 
ratio of the received signal serves as an external random 
field acting on each site, and that channel characteristics 
define the distribution of the random field. The Gallager 
and MN codes are analyzed, to find that the basic proper- 
ties of these codes remain unchanged regardless of chan- 
nel characteristics. In particular, it has been shown that 
these codes potentially saturate Shannon's limit asymp- 
totically, as K — > ao, for the Gallager code; and when 
K, L > 2 — with a few exceptions with small C and 
L values — and asymptotically as L — > oo for K = 1, 
for the MN code. Saddle-point solutions have also been 
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numerically evaluated extensively for the cases of BI- 
AWGNC and BILC channels, from which noise-overlap 
diagrams, as well as the transition and bifurcation points, 
have been characterized. 
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APPENDIX A: STABILITY OF 
PARAMAGNETIC SOLUTION FOR K > 2 

To probe the stability of paramagnetic solution, which 
exists for K > 2, we analyze the stability with respect to 
q and r only, and do not consider stability with respect to 
q and f; these conjugate variables are subsidiary to their 
counterparts, q and r, respectively, so that the former 
should not be considered as independent variables. 

Let A, B, ... denote sets of replica indices such as 
{ai ■ ■ ■ Q. m )i rn>l. We first evaluate the Hessian of the 
free energy (|32| ) with respect to 4 x (2™ — 1) variables 
{q A ,qA,r A ,r A }-- 



H 



( Hqq H qq 
H qq H qq 

o 



o 

O H r 



The block-diagonal structure of the Hessian allows us to 
decompose the stability problem into two, one with re- 
spect to q, and another with respect to r. 

Following the argument in the appendix of pp[ , one 
can say that the system is stable with respect to q if the 
matrix H r = H. 



n 



Hqq(H qq ) 



H qq is positive definite. 
This condition takes into account the fact that q depends 
on q. A corresponding statement holds for the stability 
with respect to r. 

The stability with respect to r is straightforward, by 
noting that the matrix H^-f- is negative definite, which 
means that H c = —(M/N) 2 (Hffj is positive definite. 

We consider the stability with respect to q. For K > 3, 
we have H c = [q^/C{C — 1)]/, where I is the identity 
matrix, so that the stability immediately follows, irre- 
spective of the noise level of the channel. For K = 2, the 
matrix H c is diagonal, and its A-th element is 



fa 



q 2 \r J C(C-l) 



(A3) 



Using the equality which holds under the RS ansatz, 



ro 



v m p{v) dv, 



(A4) 



(Al) where A — a% ■ ■ ■ a m , we have, as the stability condition, 



where 



H, 



ii Jab 



H, 



11) AB 

(Hii)ab 

(H r j.) j 



I AB 



I AB 



(K > 3) 

4f-V^B (K = 2) 
% \ r oJ 



Sab 
C(C- 1) 



M 
iV 



e 



TAB 



Sai 



M L(L - 1) 



5ab 



(A2) 



v m P {v)dv < (C- l)~ 1/L . (A5) 



for m = 1, . . . ,n. Since it can be shown that Ei m -\ — 
Ei m and E 2m > E 2m +2j the critical condition determin- 
ing the stability is 



Ei <{C-l) 



-l/L 



(A6) 
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